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Abstract 

The asymptotic normality of the Maximum Likelihood Estimator (MLE) is a corner¬ 
stone of statistical theory. In the present paper, we provide sharp explicit upper bounds 
on Zolotarev-type distances between the exact, unknown distribution of the MLE and its 
limiting normal distribution. Our approach to this fundamental issue is based on a sound 
combination of the Delta method, Stein’s method, Taylor expansions and conditional expec¬ 
tations, for the classical situations where the MLE can be expressed as a function of a sum 
of independent and identically distributed terms. This encompasses in particular the broad 
exponential family of distributions. 
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1 Introduction 


The asymptotic normality of maximum likelihood estimators (MLEs) is one of the best-known 
and most fundamental results in mathematical statistics. Under certain regularity conditions 
(given later in this section), we have the following classical theorem, first discussed in Fisher 
(1925). 

Theorem 1.1 (Asymptotic Normality of the MLE). Let Ai, ..., A„ be i.i.d. random variables 
with probability density (or mass) function f{xi\9), where 9 is a scalar parameter. Its true value 
is denoted as 0o- Assume that the MLE exists and it is unique and conditions (R1)-(R4), see 
below, are satisfied. Then 

(9n{X) - 9o) Af{0, 1) 

where i(0o) *5 the expected Fisher information quantity and means convergence in distribution. 

The aim of the present paper is to complement this qualitative result with a quantitative 
statement, in other words, to find the best possible approximation for the distance, at finite 
sample size re, between the distribution of Y^rei(0o) (^9n{X) — 9oJ on the one hand and Al(0, 1) 
on the other hand. In mathematical terms, for Z ~ 1), we are interested in the quantity 


dH(Vni(9o) (9n(X)-9o),zj 


= sup 
heB 


E 


h (v^rei(6»o) [9n{X) - 


-E[/i(Z)] 


( 1 . 1 ) 
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with 


= {/i ; M —>■ R, absolutely continuous and bounded} . ( 1 . 2 ) 

Distances of this type are called Zolotarev-type distances. Our main focus will lie on the classical 
situations where the MLE can be expressed as a function of a sum of independent and identically 
distributed terms. 

Consider an i.i.d. sample of observations X = (Xi,...,X„). Writing 9n{X) the MLE of 
the scalar parameter of interest 0 E 0 C R, we are interested in settings where there exists a 
one-to-one twice differentiable mapping g : 0 —>■ R such that 

q{en{X)) = (1-3) 

i=l 

for some <7 : R —>■ R. Situations of this kind are all but rare; with f{x\6) the probability density 
(or mass) function, classical examples include 

• the normal distribution with density /(x|/i, ci^) = exp (— ^{x — /r)^), x E R, for 
which E R is our unknown parameter, whereas a > 0 is considered to be known. The 
MLE ioi 6 = pL is 

1 

OniX) = -y^Xi- 
2=1 

• the normal distribution, where now the mean /r is known and 9 = represents the 
unknown parameter, with 

1 "■ 

9n{X) = -Y,{Xi-pif- 
2 = 1 

• the Weibull distribution with density f{x\a,a) = ^ (|)“ ^ exp (— {^)°‘), x >0, where cr 
is the unknown scale parameter and a > 0 is fixed. The MLE for 9 = a is defined through 

(«"(^>)“ = sE^.- 

2=1 

• the Laplace scale model with density f{x\a) = ^ exp(—|x|/cr), 0 = u > 0, over R, for 
which 

n 

9n{X) = - V|Xi|. 
n 

2=1 

Moreover, the broad one-parameter exponential families do satisfy condition (1.3); see Proposi¬ 
tion 3.1 for details. Hence, our results do apply to most of the well-known distributions. 

We now present in detail the notation and general assumptions made throughout the paper. 
We write Eg)] the expectation under the specific value 9 of the parameter. In line with the 
notation used above, the joint density or probability mass function of X is written f{x\9). The 
true, unknown value of the parameter is 9q and 0 denotes the parameter space. For Xi = Xi 
some observed values, the likelihood function is denoted by L{9-,x) = f{x\9) and we denote 
its natural logarithm, called the log-likelihood function, by l{9;x). The derivatives of the log- 
likelihood function with respect to 9 are I'{9; x),l"{9; x ),..., l^^\9‘, x), for j any integer greater 
than 2, and i{9) denotes the expected Fisher information number for one random variable. 
Whenever the MLE exists and is also unique, we will write it as before under the form 9n{X). 
For 0 being an open interval, we use the results in Makelainen et al. (1981) to secure the 
existence and uniqueness of the MLE. Thus, it suffices to assume that: 
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(Al) The log-likelihood function l{ 6 ;x) is a twice continuously differentiable function with re¬ 
spect to 9 and the parameter varies in an open interval (a, b), where a, 6 E M U {—oo, 00 } 
and a < b] 

(A2) lim l{9;x) = — 00 ; 

9^a,b 

(A3) I"{9; a;) < 0 at every point 9 E (a, b) for which I'{9; x) = 0. 

Note that we tacitly assume those conditions in Theorem 1.1 when requiring existence and 
uniqueness of the MLE. Asymptotic normality further requires the following sufficient regularity 
conditions: 


(Rl) the parameter is identifiable, which means that if 0 7 ^ 9', then : f{x\9) 7 ^ f{x\9')', 

(R2) the density f{x\9) is three times differentiable with respect to 9, the third derivative is 
continuous in 9 and f /(x|0) dx can be differentiated three times under the integral sign; 

(R3) for any 9o ^ 0 and for X denoting the support of f(xj9), there exists a positive number e 
and a function M(x) (both of which may depend on 9o) such that 


^^ogf(xl9) 


< M(x) Vx E X, 9o 


€ < 9 < 9q -j- €, 


with E 0 p[M(X)] < 00 ; 
(R4) i(9) > 0, V0 E 0. 


These conditions, in particular (R2), ensure that, provided the respective expressions exist, 
'Eiq[1'{9; X)] = 0 and \ai 0 [l'[9] X)] = ni{9). These conditions form the basis of Theorem 1.1 
above; see page 472 of Casella and Berger (2002) for a basic sketch of the proof. 

The first paper to address the problem of assessing the accuracy of the asymptotic normal 
approximation for MLE is Anastasiou and Reinert (2015). After deriving general bounds on 
Zolotarev-type distances, they use the bounded Wasserstein distance d^w, which is also known as 
Eortet-Mourier distance (see, e.g., Nourdin and Peccati (2012)) and is linked to the Kolmogorov 
distance {H is the class of indicator functions of half-spaces) via dx{■■,■) < 2a/ dbwi', ')• We state 
in Theorem 2.4 of Section 2 the bound obtained in that paper. For the broad class of distributions 
satisfying (1.3), our bound is better than, or at least as good as, the Anastasiou and Reinert 
(2015) bound (hereafter referred to as AR-bound) both in terms of sharpness and simplicity. The 
tools we use to reach this result are the Delta method, Stein’s method for normal approximation, 
Taylor expansions and conditional expectations. 

The paper is organised as follows. Our new upper bound is described, proved and compared 
to the AR-bound in Section 2. In Section 3 we then apply our results to the class of one- 
parameter exponential family distributions and treat some specific examples in detail. 


2 New bounds on the distance to the normal distribution for 
the MLE 

In order to obtain bounds on the aforementioned distance, we partly employ the following lemma. 
From now on, unless otherwise stated, || • || denotes the infinity norm || • ||oo- 
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Lemma 2.1 (Reinert (1998)). Let Yi,... ,1^ be independent random variables with E(li) = 0, 
Var(Yj) = > 0 and E \Yi\^ < oo. Let W = -^ E(1E) = 0, Var(lE) = and 

let K ~ AA(0, (T^). Then for any function h £ H, with H given in (1.2), one has 


|E[/r(lE)]-E[/r(i^)]|< 


m 

'Jn 


2 H—5-E 


a" 


i^ir 


As we shall see below, our strategy consists in benefiting from the special form of g(0ri(A^)); 
which is a sum of random variables and thus allows us to use the sharp bound of this lemma. It is 
precisely at this point that the Delta method comes into play: abusing notations and language, 
instead of comparing 9n{X) to Z ~ AA(0,1) we rather compare q{6n{X)) to Z, and then bound 
the distance between 8 n{X) and q{6n{X)). The outcome of this approach is the next theorem, 
the main result of the present paper. 


Theorem 2.2. Let Xi,...,X„ be i.i.d. random variables with probability density (or mass) 
function f{xi\9) and let Z ~ AA(0,1). Assume that (Al)-(A3) and the regularity conditions (Rl)- 
(Rf) are satisfied, and hence the MLE 9n{X) exists and is unique. Furthermore let g : 0 —)• M 
be a one-to-one twice differentiable function with q'{9) / 0 V0 E 0 and such that q (9n{X)] = 


n Y17=i9i^i)’ where the mapping g : M ^ M is such that E \g{Xi) — q{9Q)( 
the true value of the parameter. Also, there exists a positive constant 0 < e = 
with (00 — e, 00 + e) C 0. Then, for any h £ H as in (1.2) we have 


< oo for 00 E 0 
e(0o) as in (R3) 


E 


h {0n{X) - 0o))] - E [h{Z)] 


< 


\\h'\ 


n 


+ E 


2 + 1M1e 

2 ^ ^ 


9n{X) - 00 


|5(Xi)-g(0o)r 


-1{30E0: q{9)^9] + 


||/i'||^ni( 0 o) 


2 k'( 6 'o)| e-.\6-eo\< 


sup |g"( 0 )| . ( 2 . 1 ) 


Proof. The asymptotic normality of the MLE is explicitly stated in Theorem 1.1. Applying the 
widely known Delta method to this result in combination with the requirement q'{9o) 7 ^ 0 we 
obtain 

(, (4(X)) - , (9„)) ^ V( 0 .1). (2,2) 

with q ^0„(X)^ = ^ g{Xi). Using the triangle inequality we get that 


E 


< 


+ 


hi^y/ni{9o)(9n{X)- 
E h 


-E[h{Z)] 
q{9n{X))-q 


E 


Vni{9o) 

q'{Oo) 


h (Yni{9o) (9n{X) -9o)) -h 


\/nWo) 


q'{0o) 


-E[hiZ)] 

q (OniX 



(2.3) 

(2.4) 
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We first start to obtain an upper bound for (2.3) using (indirectly) Stein’s method via Lemma 2.1. 
Some simple rewriting yields 


A/nf( 6 'o) 

q'm 

1 


q{en{X)) -q 


\/ni{6o) / 1 


q'{Oo) 

1 


Y,9{Xi)-q{9,) 


where L) = {9{Xi) — q{9o)) ,i = 1,2 ,... ,n and, obviously, the li’s are independent and 

identically distributed random variables. The Central Limit Theorem applied to ^ 

plies “ E(yi)) A W(0, Var(Ti)). From (2.2) we know however that ^ 

AA(0,1); comparing the two asymptotic results reveals that, necessarily (since two normal dis¬ 
tributions can only be equal if their expectations and variances are the same), we have 


E b(^i)] = q{0o) and Var [fif(Wi)] = 


{9'{eo)f 

i{eo) 


where condition (R4) allows to divide by i{0Q). Hence, E[yj] = 0 and Var[l^] = 1 (as Lemma 
2.1 requires). Applying the result of the lemma we get 


E 


\/ni{ 0 o) 

q'iOo) 


(g {9„iX))-q{9o)) 


- mz)] 


< 


11/^11 


2 + i®liE 


\g{X,) - q{0o)\^ 


(2.5) 


Now we are searching for an upper bound on (2.4). Since the case q{9) = 9 is obvious, we from 
here on assume that q{9) / 9. To do so, we denote by 


A := A(q, 9o, X) :=/i ( i(9o) [9r,(X} - 

^/ni(9o) 


- h 


q'iOo) 


q(9n{X))-q 


and our scope is to find an upper bound for |E [A]|. Using the law of total expectation related 


to conditioning on 


9n{X) - 9o 


> e or 


9n(X) - 


|E[A]| = 


E 

+E 


< 


E 

< E 
+ E 


A 

E 

1^1 


A 


9n(X) - 9o 
n(X) - 9o 
9n(X) - 9o 
9n(X] - 9o 
9n(X) - 9o 


> € 

< e 
> e 


< e and the triangle inequality we obtain 

9n{X)-9o\ >e 
(|0„(X) - 0o| < e 
P ( 9n{X) -9q >e 


P 


|A| 9n{X) - 9o 


< e 
> e 
< e 


P 


9niX) - 9o 


< e 


P 


9niX) - 9o 
9n{X) - 9o 


> e 
< e 
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Markov’s inequality and the elementary results of P ^ 9n{X) — 9 q < < 1 and |A| < 2\\h\\ 

further yield 


E 


\E[A]\ < 2\\h\\- 


9JX) - 


+ E 


1^1 9n{X) - 9o 


< e 


( 2 . 6 ) 


We now focus on the conditional expectation on the right-hand side of (2.6). A second-order 
Taylor expansion of q ( 9n{x )) about 9o gives 


q{9nix)) = q{9o) + {9n{x) - 6lo) q'{9o) + ^ (Snix) - 9o^ q'\9*), (2.7) 


for 9* between 9n{x) and 0o- Since we assume that q'{9) / 0 V0 G 0, we can multiply both sides 
in (2.7) with • Rearranging the terms, we obtain 


y/ni{9o) (q (9n{ 


X - 


q'{9o) - Vni{9o) {9n{x) 0o) + 2q'{9of 


Using the above result along with another first-order Taylor expansion (recall that 
/n ( 9n{X) — 00 ) = op(l) as n —>■ oo) gives 


h (^^/ni{9o) (9n{x) - 0o)) - ^ (? (^n(®)) - g(0o)) 


q'{0o) 

A^^q>>^e*){Ux)-9oyh\t{x)), 


( 2 . 8 ) 


where t{x) is between Y^nf(0o) {^9n{x) — 0oj and [q {J^n{x)'j — 

combined with Lemma 2.1 in Anastasiou and Reinert (2015) related to conditional expectations 
yields 


. Equality (2.8) 


E 


|A| 9niX)-9o 


< € 


= E 


■\/ni{9o) 
2q'{9o) ' 





) h'{t{X)) 

9n{X) - 00 

< e 


^ \\h'\Wni{9o) 
~ 2|g'(0o)| 


\q"{9*)\ [9n{X)-9oy 


9n{X) - 00 


< e 


\\h'\\^/ni{9o) I 

2|9(0o)| e-.\e-eo\<e 


<-P k"w|E 

2|^i0oj| 0:|e-eo|<€ 


0n(A:) - 


0n(A:) - 


9n{X) - 00 


< e 


(2.9) 


Combining the bounds in (2.5), (2.6) and (2.9) gives the result of the theorem. 
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Remark 2.3. (1 
behaviour of E 


The convergence of the second and third terms is governed by the asymptotic 
On{X) — Oq] , whose rate of convergence is O (-)• This result is obtained 


using the decomposition 


E 


(OniX) - e^f = Var[0„(X)] + bias2[4(X)]. 


( 2 . 10 ) 


Under the standard asymptotics (from the regularity conditions (R1)-(R4)) the MLE is asymp¬ 
totically efficient, meaning that 

nVar[0„(X)] -^ [i(0o)]"\ 

n—^■oo 

and hence the variance of the MLE is of order -. In addition, from Theorem 1.1 the bias of 
the MLE is of order see also Cox and Snell (1968), where no explicit conditions are given. 
Combining these two results and using (2.10) shows that the mean squared error of the MLE is 
of order -. 

n 

(2) In the simplest possible situation where 6 n{X) is already a sum of i.i.d. terms, q{x) = x and 
hence our upper bound simplifies to 


E 


h{^ni{eQ)[en{x)- 


-E[L(Z)] 


<M ('2 + [f(0o)]tE[|<7(Xi)-0o|' 

Vra V L 


which is equivalent to Lemma 2.1. 

In order to appreciate the sharpness and simplicity of our bound (2.1), we compare it to the 
AR-bound. To this end, we now state the main result of Anastasiou and Reinert (2015). 

Theorem 2.4 (Anastasiou and Reinert (2015)). Let Xi, X 2 , ■ ■ ■, X^ be i.i.d. random variables 
with density or frequency function f{xi\9) such that the regularity conditions (R1)-(R4) are satis- 

r 3 ' 

fied and that the MLE, 6n{X), exists and it is unique. Assume that E (^log/(Xi|0))g^g^ < 


00 and that E 


9n{X)-9oy 


< 00 . Let 0 < e = e{9o) be such that {9o — e,9o + e) C Q as in 


(R3) and let Z ~ AA(0,1). Then for any function h that is absolutely continuous and bounded, 

-E[h{Z)] 


E 


h iy/niido) {9niX) - 


< 


\\h'\ 


n 


2 + 


-E 


E 


+ 2 


[*(^o )]2 

^iX) - 9o) 




+ 


\\h'\ 


E 


1 

+ 

E 

sup 

l^^\9;X) 

r 

2 


\e:|0-6»o|<e 


J 


^/ni{9o) 

\9n{X)-9o\<e 


\R2i9o-,X)\ \9n{X)-9o\ <e 



2 



1 

2 



E 

[9n{X) - 9o) 








where 


R2{9 o; x) = {9n{x) - 9 q) {l"{9o] x) + n i{9o)) . 


( 2 . 11 ) 
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Obvious observations are that the AR-bound requires finiteness of the fourth moment of 
9n{X) — Oq and that this bound is more complicated than ours. Let us now comment on the 
bounds term by term. 


• In the first term of the bounds, the different positioning of the expected Fisher information 
number is explained by the fact that we apply Lemma 2.1 to the standardised version of 
g{Xi)^g{X 2 )i... ^g{Xn)i which have variance , while Anastasiou and Reinert (2015) 

obtain the result by applying the lemma after standardising l'{9o\ Xi),l'{9 q]X2)., ■ ■ ■, V{9q; Xn), 
which have variance equal to f(0o)- 


• The second and third terms vanish in our bound when q{9) = 9\/9 £ Q, while the AR- 
bound does not take this simplification into account. In addition, when q(9) ^ 9 the 
second term is the same in both bounds, whereas the third term in our bound reads 


E 


(9n{X) - 0 o) 


2 


\\h'\\^ni{9Q) 

2k'(0o)| 


sup \q"{9)\ 
e:\e-eo\<e 


( 2 . 12 ) 


and is to be compared to 

\\h'\\ 

2^yni{9o) 

\\h'\\ 



’ / 



E 

sup 

l^^\9]X) 



\e-.\e-eo\<e 


J 


+ 


Vni{9o) 


:E 


\9niX)-9o\<e 
\R2{9 o;X)\ \9n{X)-9o\<e 


E 


.(^)- 0 o)^ 


(2.13) 


where R2{9 o]x) = {9n{x) — 9 q) {I” {9o; x) + n i{9Q)). The second derivative, q''{9), plays 
in our bound the role of l^^\9;X), up to an important difference: l^^\9]X) is a sum. 
Consequently, the first term in (2.13) has ^/n in its numerator, exactly as in (2.12). The 
distinct positioning of the information quantity f(0o) has the same reason as explained 
above. Besides the obvious additional term in the AR bound (the second term in (2.13)), 
our bound is also clearly sharper at the level of moments of 9n{X) — 9 q since 


E 


9n{X) - 00 


< 


E 


k{X) - 00 


1 . 

2 


by the Cauchy-Schwarz inequality. 

Erom this comparison one sees that our new bound is simpler and, moreover, has one term less. 
This is particularly striking in the simplest possible setting where 9n{X) is a sum of i.i.d. terms, 
where our bound clearly improves on the AR-bound. An advantage of the AR-bound is its wider 
applicability as it works for all MLE settings, even when an analytic expression of the MLE is 
not known. 


3 Calculation of the bound in different scenarios 

In this section we shall consider different examples for which we explicitly calculate our upper 
bound from Theorem 2.2 and compare it to the AR-bound. To further assess its accuracy, we 
simulate data from various distributions and compare our bound to the actual distance between 
the unknown exact law of the MLE and its asymptotic normal law, for distinct values of the 
sample size n. 































3.1 Bounds for one-parameter exponential families 

The probability density (or mass) function for one-parameter exponential families is given by 

fix\9) = exp{k{9)T{x) - A{9) + S{x)} (3.1) 

where the set B = {x : f{x\9) > 0} is the support of the density and does not depend on 9] 
k{9) and A{9) are functions of the parameter; T{x) and S{x) are functions only of the data. 
Whenever k{9) = 9 we have the so-called canonical case, where 9 and T{X) are called the natural 
parameter and natural observation (Casella and Berger, 2002). The identifiability constraint in 
(Rl) entails that k'{9) ^ 0 (Geyer, 2013), an important detail for the following investigation. 

Proposition 3.1. Suppose Xi,...,X„ are i.i.d. with probability density (or mass) function 
that can be expressed in the form of (3.1). Assume that (Al)-(A3) and the regularity conditions 
(R1)-(R4) are satisfied, and hence the MLE 9n{X) exists and is unique. Let 9 i—)• D{9) = 
be invertible. Then q{-) = D{-), with g : 0 —)• M as in Theorem 2.2. 

Proof. Using (3.1), we have that 

n f n n ^ 

L{9; ®) = ]^ f{xi\9) = exp < k{9) ^ T{xi) — nA{9) -|- E »(*•) • 

i=l I i=l i=l ) 

n n 

li9-,x) = k{9)'^T{xi) - nA{9) + Esfe). 

i=l i=l 

and hence 

n 1 

l'(9;x) = k'(e)y"T(xi)-nA'(e) = 0^0(9) = -Vrfe), 

n 

i=l i=l 

which means that 9n{X) = D~^ („ under the invertibility assumption for D{9). 

The claim readily follows. ■ 

This result hence shows that, as announced in the Introduction, the broad one-parameter ex¬ 
ponential families do satisfy (1.3). Consequently, Theorem 2.2 can be applied to (3.1), resulting 
in 


Corollary 3.2. Let Xi,... ,Xn be i.i.d. random variables with the probability density (or mass) 
function of a single-parameter exponential family. Assume that (A1)-(A3) and (R1)-(R4) are 
satisfied, and hence the MLE 9n{X) exists and is unique. With Z ~ AA(0, 1), h ^ H as defined 
in (1.2) and 0 < e = e{9o) : {9o — e,9o + e) C Q as in (R3), we obtain 


E 


h - 0o))] - E [h{Z)] 


< 


m 


n 


2 + 


\k'{9o)fE \T{Xi) - D{9o)f 


+ E 


9n{X) - 


\A"{9o)-k"{9o)Di9o)\-^ 
2 ' 


1{30 E 0 : D{9) ^ 9] 




\\h'\\V^\k'{9o)\ 


2^\A"{9o)-k”{9o)D{9o)\ e-.\e-eo\< 


sup In" (0)1 
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Proof. We readily have 


i{eo) = E [-l"{eo;Xi)] = A"{ 9 o) - k"{eo)E[T{X^)] 


A"{eo)k'{eo)-k"{eo)A'{9o) 

k'{9o) 


and g'(6*o) = ■ Combining these two results, 

[k (c^o)J 

Vm _ _ \k'{9o)\l _ \k'{9o)\ 

Wi^o)\ ^\A"{9o)k'{9o)- k"{9o)A'{9o)\ y^\A"{9o) - k"{9o)D{9o)\' 

This result, along with the fact that g{x) = T{x) and q{9) = D{9) by Proposition 3.1, allows to 
deduce the announced upper bound from Theorem 2.2. ■ 


Remark 3.3. It is particularly interesting to spell out this bound in the canonical case k{9) = 9. 
Since then k'{9) = 1, D{9) = A'(9), we find 


E 


h{y'ni{9o){9n{X)-9o)) -E[h{Z)] 


< 


\w\ 


E 


n 


2 + 


\TiX,)-Ai9oW 


\A"{9o)^‘- 


+ E 


9n{X) - 9o 


+ 


l{39 e & : A'{9) t^9} 

snp 

2v^|^"(6»o)| e-.\e-eo\<e 



As i{9) = A"[9) and I"[9] X) = —nA"(9), R2(9; x) = 0 and straightforward manipulations show 


that all terms in the AR-bound coincide with those in our bound, except for 


E 


0n(X) - 9o 


making the AR-bound less sharp than ours. However, Anastasiou and Reinert (2015) have 


shown that, in the canonical exponential setting, their bound can actually have an E 


OniX) - 


factor, implying that both bounds are exactly the same in the canonical case. In order to get 
an idea of how our bound improves on the AR-bound in non-canonical cases, we treat the 
exponential distribution under a non-canonical parametrisation in Subsection 3.3. 


3.2 Bounds for the Generalised Gamma distribution 


Let us consider Xi,..., X^ i.i.d. random variables from the Generalized Gamma GG{9,d,p) 
distribution, where the shape parameters d,p > 0 are considered to be known and the scale 
parameter 9 is the unknown parameter of interest. The Generalised Gamma distribution includes 
many other known distributions as special cases: the Weibull for d = p, the Gamma when p = 1, 
and the negative exponential when p = d = 1. Indeed, with r(-) denoting the Gamma function, 
the probability density function for x > 0 is 


fix\ 0 ) 


px'^-iexp{- (f)^} 



f xP 

exp s — 7 —h logp — dlog 6* -|- (d — 1) log x 


log T 
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where, in the terminology of one-parameter exponential families, B = (0,oo), 0 = (0, oo), 
T{x) = x^, k{6) = — A{9) = dlogO and S{x) = logp -|- (d — 1) logx — log Simple 

steps yield 


^ n / n 

l{Oo;x) = +nlogp -ndlogOo + {d - l)log 


^ i=l 


\i= 


n j / n \ 


^0 i=l 


Xi I - n log ( r ( ^ 


2=1 


It is easy to verify that indeed l"{6n{x);x) = < 0, which shows that the MLE exists 

and is unique. The regularity conditions (R1)-(R4) are also satisfied and using Corollary 3.2 for 

w\ 


e = ^ we obtain 


E 


h{y'ni{9o){en{X)-eo)) -E[h{Z)] 


< 


2+ (3 + 6 ^ 

n \ V a 


-p / nd+l 

+ 11-2(^)^ 

p [ 21“ 

P 


+ 


ndJ p ^nd 

\\h'\\^/ndp\p — 1 | / 1 


2 Y ( ^^+2 
P 


r f 

p 






p-2 


l{p > 2} 


(3.2) 


Let us briefly show how to obtain this bound. For the Generalised Gamma distribution, 

3l 


D{9o) = q{9o) = pi and thus E \T{X) - D{9o)\^ 


= E 


XP - !^9l 

P ^ 


This third absolute 


moment is very complicated to calculate. Therefore, we use Holder’s inequality and the fact 
X ~ GG(0O) d,p) ^ XP ~ Gamma to get 


E 

- (!«s 

3' 

< 

E 



p 




\ p j 


3 

4 


E + 

^4p d 


0 - \ 6-1-3- 


d 

i- 

p \ p 

Simpler calculations yield 


p) 

3 


+ 6 


< 


^ ) E [.Y=f] - 4 


d9l 


.d. 


f) E [X^] - 4-0PE 




wmi 


Vp 


^\A"{9o)-k"{9o)D{9o)\ Vd9l' 


(3.3) 


(3.4) 
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Using that Xf ~ Gamma ^ Yll=i ~ Gamma (ji^, , we get 


E 


0n(X) - 01 


= 

\nd 


E 


E^f 


2 n 

p 


\i=l 


+ 0, 




«oE 


E-^f 


j^-| 

P 


V 2=1 


Vnd 


2 r 


/ nd-\-2 \ 


= 02 ^ V P / ^ 02 _ ^^2 


■p / nA 
P 


+ 0n “ 20n 


Vnd 


1 r 


nd+1 

P 


■p / nd 
P 


= eUi 




1 r 


+ 


\nd 


1 r 


(“F) 


(3.5) 


Regarding sup \D"{9)\, one has to be careful as the supremum depends on the value of p: 
e-.\0-eo\<e 

sup |d(p — = d|p — 1| sup |0^“^| 

0-.\e-eo\<t " ' 


0-.\0-0o\<e 

(0o-eF"^ if0<p<2 


= d\p — 1| < 


(3.6) 


_(0o + eY Y if p > 2. 


Thus, applying now the results of (3.3), (3.4), (3.5) and (3.6) on the general expression for the 
upper bound in Corollary 3.2, we obtain the result in (3.2). 

Remark 3.4. (1) The bound in (3.2) is O This is not obvious as we need to comment 

-pf nd-\-l \ 2 Y'f \ 

^ ~ ^ {Ya) ^ ^ j ' following Taylor 

expansion for a ratio of Gamma functions (see Tricomi and Erdelyi (1951)) 


r(z + a) ^ fe / (a-b)(a + 5- 1) 

r(z + 6) \ 2z 


(i-n) 


for large z (here, nd/p) and bounded a and 6, we can see that this term is of order leading 
to the overall order of Y=. 

Vn 

(2) The indicator function in (3.2) comes from the fact that q{6) = 0V0 G 0 <(=^ d,p = 1. 


3.3 Bounds for the (negative) exponential distribution 

In this subsection, we consider the most famous special case of the Generalised Gamma dis¬ 
tribution: the (negative) exponential distribution. First we will treat the canonical form of 
the distribution and then we will change the parameterisation to discuss the more interesting 
non-canonical setting. 


3.3.1 The canonical case: Exp(0) 

We start with Xi,..., X„ exponentially distributed i.i.d. random variables with scale parameter 
0 > 0 and probability density function f{x\6) = 0exp{—0x} = exp{log0 — 0 x} for x > 0, which 
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we write Exp{9). In terms of (3.1), this means B = (0, oo), 0 = (0, oo), T{x) = —x, k{6) = 0, 
AiO) = — log 9 and S{x) = 0. Further we have that 


l'{9■x) = '^-Y,x^, l''{9- 


X = - 


i=l 


n 


the unique MLE is given by 9n{X) = 1/X with X = - XlILi are satished. 


With this in hand, we can easily see that D{9o) = q{9o) = ~ ^ do 

\k'{9o)\ 


_ A'{0o) _ 1 _ j 

~ ~~ 

= 9o. 


(3.7) 


^\A"{9o) - k"i9o)D{9o)\ 

Simple calculations allow us here to bypass the Holder inequality used for the Generalized 
Gamma case and to obtain E [|r(X) — Z)(0o)P] = E ^—X < Since W ~ 

Exp(0), Vi G {1,2, ...,n} then X ~ Gam(n,n0), with Gam(a,/3} being the Gamma distri¬ 
bution with shape parameter a and rate parameter /?. Consequently, 

(n(9o)2 2n9l . {n + 2)9l 
+ ^0 = 


E[{9niX) - 9or] = 


(n — l)(n — 2) n — 1 


(n — l)(n — 2) 


Moreover, for e > 0 such that 0 < e < 0O) we obtain sup \D"{9)\ = . Choosing e = ^, 

e:\9-eo\<e ^ 


we get sup \D''{9)\ = Using this result and (3.7), Corollary 3.2 gives 

9-.\e-eo\<e 0 


E 


h{y^ni{9o)(9n{X)-9o)) -E[h{Z)] 


<4.41456^ +8\\h\\ 


(n + 2) 


+ 8||h'| 


^/n (n — l)(n — 2) 

^/n{n + 2) 

(n — l)(n — 2) ’ 


(3.8) 


This bound is of order O and coincides, as discussed in Remark 3.3, with the AR-bound. 


3.3.2 The non-canonical case: Exp(|) 


We now proceed to examine the more interesting case where Xi,..., Xn are i.i.d. random 
variables from Exp (^). The probability density function is 


1 


9 


1 


9 


f{x\9) = -rexp { ~ 7 ;^ f ~ 1 ~log0 — -x 


1 


9 


corresponding to R = (0, oo), 0 = (0,oo), T{x) = x, k{9) = — A{9) = log0 and S(x) = 0. 
As before, simple steps give that the MLE exists, it is unique and equal to 9n{X) = X. The 
regularity conditions are satished and for e = ^ we obtain using Corollary 3.2 that 


E 


h{y'ni{9o){9n{X)-9o]] -E[h{Z)] 


<4.41456 


\\h'\ 


n 


(3.9) 


Indeed, D[9q) = (?(0o) = making the last two terms of the bound in Corollary 3.2 vanish. 


The result then follows from E 


\T{X)-D{9of 

W{9o)\ 


= E 


|A - 


< 2.414566l|] and 


^\A"{9o)-k"{9^)D{9^)\ 


1 
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Remark 3.5. (1) The order of the bound in terms of the sample size is, as expected, 
corresponding to the order obtained for the Generalized Gamma distribution. The constant 
here is better than the one inherited from (3.2) for p = d = 1, thanks to a sharper bound for 
E[\TiX)-Dieo)f . 

(2) The AR-bound is given by 

w 


4.41456- 


+ 8 
n n 


\\h\\ , Jh^ 


+ 2 - 


n 


+ 80 


\w\ 


n 


^ + 3 
n 


1/2 


showing that our new bound is an improvement. 


3.3.3 Empirical results 

For a more complete picture, we also assess the accuracy of our results using simulation-based 
data. The process we follow is quite simple. We generate 10000 trials of n = 10,100,1000,10000 
and 100000 random i.i.d. observations from the exponential distribution Exp(^) (non-canonical 

case). As function h we choose h{x) = with h & H, \\h\\ = 0.5 and \\h'\\ = 

Simple calculations yield E[h{Z)] = 0.379 and each trial gives an MLE 9n{X), hence we have 
10000 empirical values of h (-^ni(0o) ( 0n{X) — 0o) ) to compare to 0.379. Taking the average 


hence provides a simulated estimation of 


E 


h 


— E[h{Z)] , which we 


'ni{9o) [9n{X)-9o^ 

compare to the upper bound given in (3.9). Our bound provides a very strong improvement 
on the AR-bound (see Table 3.1). Of course, this estimated distance is only a lower bound to 
the true distance, as we have chosen a particular function h instead of the supremum over all 
functions h ^ H, but its calculation still provides an idea of the accuracy of our bounds. This 
closeness logically increases with the sample size and becomes quite sharp for n > 100. 


Table 3.1: Simulation results for the Exp(^) distribution treated as a non-canonical exponential 
family 


n 

E h[,/^iA(^i9niX)-9o)) -E[h{Z)] 

New Bound 

AR-bound 

10 

0.0034 

0.321 

11.888 

100 

0.0022 

0.101 

3.401 

1000 

0.0012 

0.032 

1.058 

10000 

0.0008 

0.010 

0.333 

100000 

0.0004 

0.003 

0.105 
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